Information processing device, information processing method, and program

ABSTRACT

An information processing device includes: a storage unit that stores first information indicating at least one of a structure or a physical property of a first object obtained by capturing a real physical body into a virtual space; and a specifying unit that specifies an arrangement of a second object indicating a virtual object in the virtual space so that the arrangement is capable of expressing an interaction with the first object, based on an arrangement condition of the second object and on the first information.

FIELD

The present disclosure relates to an information processing device, aninformation processing method, and a program.

BACKGROUND

Patent Literature 1 discloses a technique of acquiring athree-dimensional object model corresponding to text display from athree-dimensional object model database and transforming thethree-dimensional object model based on an attribute value identified bya text analyzer.

CITATION LIST Patent Literature

Patent Literature 1: JP 5908855 B2

SUMMARY Technical Problem

The above-described conventional technology includes a technology ofcapturing a measured real environment into virtual reality (VR) andproviding, to the user, an image obtained by combining an object withthe virtual reality. However, the conventional technology has adifficulty in reflecting information such as mass, rigidity, part, andthe like lost in the measurement of the real environment to the virtualreality, leading to an occurrence of a gap between the virtual realityobject and the object to be combined in some cases.

In view of this, the present disclosure provides an informationprocessing device, an information processing method, and a programcapable of suppressing strangeness or incompatibility regarding anobject displayed in virtual reality into which a measured realenvironment has been captured.

Solution to Problem

To solve the problems described above, an information processing deviceincludes: a storage unit that stores first information indicating atleast one of a structure or a physical property of a first objectobtained by capturing a real physical body into a virtual space; and aspecifying unit that specifies an arrangement of a second objectindicating a virtual object in the virtual space so that the arrangementis capable of expressing an interaction with the first object, based onan arrangement condition of the second object and on the firstinformation.

Moreover, an information processing method executed by a computerincludes: storing, in a storage unit that stores first informationindicating at least one of a structure or a physical property of a firstobject obtained by capturing a real physical body into a virtual space;and specifying an arrangement of a second object indicating a virtualobject in the virtual space so that the arrangement is capable ofexpressing an interaction with the first object, based on an arrangementcondition of the second object and on the first information.

Moreover, a program causes a computer to execute: storing, in a storageunit, first information indicating at least one of a structure or aphysical property of a first object obtained by capturing a realphysical body into a virtual space; and specifying an arrangement of asecond object indicating a virtual object in the virtual space so thatthe arrangement is capable of expressing an interaction with the firstobject, based on an arrangement condition of the second object and onthe first information.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of a configuration of adisplay system including an information processing device according to afirst embodiment.

FIG. 2 is a diagram illustrating an example of an outline of theinformation processing device according to the first embodiment.

FIG. 3 is a flowchart illustrating an example of a processing procedureexecuted by the information processing device according to the firstembodiment.

FIG. 4 is a diagram illustrating an example in which the informationprocessing device recognizes a physical body.

FIG. 5 is a diagram illustrating an example of a physical bodyrecognition model.

FIG. 6 is a diagram illustrating an example of a structure/physicalproperty model.

FIG. 7 is a flowchart illustrating an example of a processing procedureof missing defect detection executed by the information processingdevice according to the first embodiment.

FIG. 8 is a diagram illustrating an example in which the informationprocessing device detects a missing defect in a physical body.

FIG. 9 is a diagram illustrating an example in which the informationprocessing device complements a missing defect in a physical body.

FIG. 10 is a flowchart illustrating an example of a processing procedureof interaction estimation executed by the information processing deviceaccording to the first embodiment.

FIG. 11 is a diagram illustrating an example in which the informationprocessing device evaluates an interaction between physical bodies.

FIG. 12 is a flowchart illustrating an example of a processing procedureof specifying an interaction executed by the information processingdevice according to the first embodiment.

FIG. 13 is a diagram illustrating an example in which the informationprocessing device specifies an arrangement of a physical body.

FIG. 14 is a diagram illustrating an example in which the informationprocessing device expresses an interaction between a physical bodyobject and an object.

FIG. 15 is a diagram illustrating an example of a display systemaccording to a second embodiment.

FIG. 16 is a flowchart illustrating an example of a processing procedureexecuted by an information processing device according to the secondembodiment.

FIG. 17 is a diagram illustrating an example of a display systemaccording to a third embodiment.

FIG. 18 is a flowchart illustrating an example of a processing procedureexecuted by an information processing device according to the thirdembodiment.

FIG. 19 is a hardware configuration diagram illustrating an example of acomputer that actualizes functions of an information processing device.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure will be described below in detailwith reference to the drawings. In each of the following embodiments,the same parts are denoted by the same reference symbols, and arepetitive description thereof will be omitted.

First Embodiment

[Outline of Display System According to First Embodiment]

FIG. 1 is a diagram illustrating an example of a configuration of adisplay system including an information processing device according to afirst embodiment. A display system 100 illustrated in FIG. 1 includes,for example, a head mounted display (HMD), a smartphone, a game machine,and the like. For example, the display system 100 provides a user withan image of virtual reality (VR), live-action VR, augmented reality(AR), and the like. The image includes, for example, a moving image, astill image, and the like. The following will describe an exemplary casewhere the display system 100 provides a live-action VR image to theuser. For example, the live-action VR captures a real environment into avirtual space by measurement, and provides a three-dimensional imageobtained by combining an object with the virtual space.

For example, in the live-action VR, when information such as mass,rigidity, and part of a physical body is lost at measurement of the realenvironment, there is a possibility of occurrence of a gap between anobject to be combined with virtual reality and the measured actualphysical body. The real environment is, for example, a real environmentto be reproduced as a virtual space. Therefore, it is desired, in thelive-action VR, to suppress a gap between an object to be combined withvirtual reality and a measured actual physical body by reflectinginformation such as mass, rigidity, part, and the like lost inmeasurement of the real environment onto virtual reality as much aspossible.

FIG. 2 is a diagram illustrating an example of an outline of aninformation processing device 30 according to the first embodiment. Asillustrated in FIG. 2, the information processing device 30 estimates aphysical body OB, a structure ST, and a property Q from informationobtained by measuring a real environment P. The information processingdevice 30 arranges an object C in a virtual space V based on anestimation result, thereby providing the user with the virtual space Vobtained by integrating the real environment P with the object C. Theobject C is an example of a second object.

Returning to FIG. 1, the display system 100 includes a sensor unit 10, adisplay device 20, and the information processing device 30. Theinformation processing device 30 is configured to be communicable withthe sensor unit 10 and the display device 20.

The sensor unit 10 includes various sensors and the like that measurethe real environment. The sensor unit 10 includes, for example, animaging device (sensor) such as a time of flight (ToF) camera, an RGBcamera, a stereo camera, a monocular camera, an infrared camera, a depthcamera, and other cameras. The sensor unit 10 includes, for example, asensor such as an ultrasonic sensor, a radar, a light detection andranging or laser imaging detection and ranging (LiDAR), or a sonar. Thesensor unit 10 supplies measurement information measured by the sensorto the information processing device 30.

The display device 20 has a function of displaying various types ofinformation. The display device 20 is controlled by the informationprocessing device 30. The display device 20 includes, for example, adisplay device that displays various types of information. Examples ofthe display device include a liquid crystal display (LCD) device, anorganic light emitting diode (OLED) device, and a touch panel.Furthermore, the display device 20 according to the present embodimentmay output information by using a projection function.

[Configuration of Information Processing Device According to FirstEmbodiment]

The information processing device 30 is a dedicated or general-purposecomputer, for example. The information processing device 30 includes astorage unit 31 and a control unit 32. For example, the informationprocessing device 30 may be incorporated in the same housing as at leastone of the sensor unit 10 or the display device 20. The control unit 32of the information processing device 30 is electrically connected to thestorage unit 31.

The storage unit 31 stores various data and programs. The storage unit31 is implemented by a semiconductor memory element such as a randomaccess memory (RAM) or a flash memory, or a storage device such as ahard disk or an optical disk, for example. The storage unit 31 storesfirst information 31A indicating the structure and physical propertiesof a physical body object obtained by capturing a real physical bodyinto the virtual space V. The physical body object is an example of afirst object. The physical body object indicates a physical bodyobtained by being captured from the real environment into the virtualspace V, for example. In the following description, the physical bodyobject may be simply referred to as a physical body. The storage unit 31stores map information 31M obtained by measuring the real environment.The map information 31M includes information regarding the realenvironment, such as a three-dimensional shape, color information,position information for each physical body, category information, andthe like.

The storage unit 31 stores information such as a physical bodyrecognition model 311, a structure/physical property model 312, astructural condition database (DB) 313, a 3D model DB 314, an object DB315, and an interaction DB 316, for example. The physical bodyrecognition model 311 includes data indicating a model for recognizing aphysical body subjected to machine learning, for example. Thestructure/physical property model 312 has data indicating a model forrecognizing the structure and physical properties of a physical body,for example, the structural condition DB 313 includes data indicating astructural condition for recognizing a physical body subjected tomachine learning, for example. The 3D model DB 314 has informationindicating the shape, structure, physical properties, and the like ofthe physical body subjected to machine learning, for example. The 3Dmodel DB 314 is constructed using, for example, 3D modeling software orthe like. The object DB 315 includes, for example, data indicating thestructure and physical properties of the object C. The object C is anexample of a second object. The object DB 315 includes secondinformation 31B indicating the characteristic of the object C. Theinteraction DB 316 includes data indicating an arrangement condition 31Cof the object C. In the interaction DB 316, for example, the arrangementcondition 31C indicating an interaction to be performed by the object Cis set by a game designer or the like. The arrangement condition 31Cincludes, for example, an arrangement condition 31C of the object C,such as “sitting down”, “standing up”, “lying”, “reclining”, and thelike.

In the present embodiment, there is no need to store all of the physicalbody recognition model 311, the structure/physical property model 312,the structural condition DB 313, the 3D model DB 314, the object DB 315,and the interaction DB 316 in the storage unit 31, and these may bestored in an information processing server, a storage device, or thelike accessible from the information processing device 30, for example.

The control unit 32 includes functional units such as a measurement unit321, a first recognition unit 322, a second recognition unit 323, amissing defect detection unit 324, an estimation unit 325, a specifyingunit 326, a processing unit 327, and a display control unit 328. In thepresent embodiment, the control unit 32 further includes functionalunits such as a missing defect complementing unit 324A and a correctionunit 325A. Each of the functional units of the control unit 32 isimplemented by execution of programs stored in the informationprocessing device 30 by a central processing unit (CPU), a micro controlunit (MCU), or the like, using random access memory (RAM) or the like,as a working area. In addition, each of the functional units may beimplemented by an integrated circuit such as an application specificintegrated circuit (ASIC) or a field-programmable gate array (FPGA).

The measurement unit 321 measures a real physical body provided in thereal environment P based on the sensor information of the sensor unit10. The measurement unit 321 measures a geometric shape in the realenvironment P using a known three-dimensional measurement technique, forexample. Examples of applicable three-dimensional measurement techniquesinclude techniques such as ToF and Structure-from-Motion. Themeasurement unit 321 supplies measurement information indicating ageometric shape, a position, and the like in the real environment P tothe first recognition unit 322. The measurement unit 321 stores themeasurement information in the storage unit 31 as the map information31M of the real environment P.

The first recognition unit 322 recognizes a physical body in the realenvironment P based on the measurement information from the measurementunit 321. For example, the physical body recognition model 311 includesa plurality of models such as a sofa, a chair, a window, a television, atable, a desk, a mat, a human, and an animal. In this case, the firstrecognition unit 322 searches for a model that matches or resembles thegeometric shape indicated by the measurement information from among themodels of the physical body recognition model 311, and recognizes thephysical body in the real environment P as a physical body object basedon the model. The first recognition unit 322 supplies the recognitionresult to the second recognition unit 323.

The second recognition unit 323 recognizes the structure, physicalproperties, and the like of the physical body object recognized by thefirst recognition unit 322. For example, the structure/physical propertymodel 312 has a model that associates the above-described model with thestructure and physical properties. For example, the second recognitionunit 323 searches for a model that matches or resembles the recognizedphysical body object from among the models of the structure/physicalproperty model 312, and recognizes the structure and physical propertiesindicated by the model as the structure and physical properties of thephysical body. The second recognition unit 323 generates the firstinformation 31A indicating a recognition result, and stores thegenerated first information 31A in the storage unit 31 in associationwith the recognized physical body object. Note that the secondrecognition unit 323 is an example of a recognition unit, and the firstrecognition unit 322 may be included in the configuration.

The missing defect detection unit 324 detects a structural missingdefect in the recognized physical body object. For example, in a casewhere the sensor unit 10 measures the real environment P, there is acase where it is difficult to measure the entire shape of the physicalbody due to the measurement angle and the positional relationshipbetween the physical bodies. The missing defect detection unit 324detects a missing defect in the physical body based on the structuralcondition of the physical body provided in the structural condition DB313. The structural condition of a physical body includes, for example,a condition for recognizing a structure such as components of thephysical body and a positional relationship of the components. Forexample, when the physical body is a chair, the components of thephysical body are required, as a condition, to have a structure having aseat and a plurality of legs. The missing defect detection unit 324performs physical simulation on the recognized physical body to detect amissing defect, safety, or the like of the physical body. The physicalsimulation is, for example, a program for confirming behavior andstability of a physical body. The missing defect detection unit 324supplies a detection result to the specifying unit 326.

When the missing defect detection unit 324 has detected a missingdefect, the missing defect complementing unit 324A changes the firstinformation 31A to complement the missing defect. The missing defectcomplementing unit 324A recognizes a missing defect portion of thephysical body object based on data such as the shape, structure, andphysical properties of the 3D model (physical body) included in the 3Dmodel DB 314, for example, and complements the missing defect portion.After having complemented the missing defect, the missing defectcomplementing unit 324A adds information corresponding to thecomplemented portion to the first information 31A.

The estimation unit 325 estimates an interaction between the pluralityof recognized physical bodies. For example, the estimation unit 325specifies the positional relationship between the recognized physicalbodies and estimates an interaction between the physical bodies based onthe first information 31A for each of the physical bodies. In a casewhere the interaction between the physical bodies has been successfullyestimated, the estimation unit 325 evaluates the interaction. Forexample, the estimation unit 325 evaluates a difference in the degree ofdeformation depending on the presence or absence of interaction.Specifically, the estimation unit 325 searches the 3D model DB 314 for amodel resembling the physical body object, and evaluates the degree ofdeformation of the physical body object by using the shape of a portionhaving no interaction in the extracted model. In addition, in a casewhere the interaction between the physical bodies has not beensuccessfully estimated, the estimation unit 325 does not change thefirst information 31A.

The correction unit 325A corrects the first information 31A regardingthe physical body object based on the result of evaluating the degree ofdeformation of the physical body by the estimation unit 325. In a casewhere information such as the amount of deformation is obtained in theevaluation of the degree of deformation of the physical body object, thecorrection unit 325A corrects the first information 31A so as toapproach the information. With this configuration, in a case where aninteraction occurs between physical bodies, the correction unit 325A canreflect the interaction onto the first information 31A.

The specifying unit 326 specifies an arrangement of the object C in thevirtual space V so that the arrangement is capable of expressing theinteraction with the physical body object based on the arrangementcondition 31C of the object C and on the first information 31A. Forexample, the capability of expressing the interaction represents acapability of displaying the object C according to the interactionbetween the physical bodies. For example, the specifying unit 326searches the virtual space V for the optimum arrangement of the object Cso as to satisfy the arrangement condition 31C of the object C. Thespecifying unit 326 stores the specified result in the storage unit 31.

For example, the specifying unit 326 specifies an arrangement of theobject C in the virtual space V so that the arrangement satisfiesphysical conditions between a part of the object C and the physical bodyobject corresponding to the arrangement condition 31C and so that thearrangement is capable of expressing the interaction with the physicalbody object. For example, the specifying unit 326 specifies anarrangement of the object C in the virtual space V so that thearrangement is capable of expressing the interaction with the physicalbody object based on the positional relationship between the object Cand the physical body object in the virtual space V.

The specifying unit 326 specifies an arrangement of the object C in thevirtual space V so that the arrangement is capable of expressing theinteraction with the physical body object based on the secondinformation 31B of the object DB 315, the arrangement condition 31C ofthe object C, and the first information 31A.

In a case where it is not possible to specify the arrangement of theobject C in the virtual space V so that the arrangement is capable ofexpressing the interaction with the physical body object, the specifyingunit 326 specifies the arrangement of the object C based on anotherarrangement condition 31C different from the arrangement condition 31C.

In a case where it is not possible to specify the arrangement of theobject C in the virtual space V so that the arrangement is capable ofexpressing the interaction with the physical body object, the specifyingunit 326 does not arrange the object C in the virtual space V. With thisconfiguration, the information processing device 30 can prevent anoccurrence of a gap between the physical body object and the object C.

The processing unit 327 executes a process of expressing an interactionbetween the physical body object and the object C based on thearrangement of the object C in the virtual space V specified by thespecifying unit 326 and the first information 31A. For example, theprocessing unit 327 executes physical simulation based on the structure,physical properties, and the like of the physical body, therebyexpressing the interaction between the physical body object and theobject C. The processing unit 327 stores the processing result in thestorage unit 31.

The display control unit 328 performs control to create a VR imagereflecting the processing result of the processing unit 327 and displaythe VR image on the display device 20. The display control unit 328instructs the display device 20 to display the VR image. As a result,the display device 20 displays the VR image expressing the interactionbetween the physical body object obtained by capturing the realenvironment into the virtual space, and the object C.

The functional configuration example of the information processingdevice 30 according to the present embodiment has been described asabove. The above configuration described with reference to FIG. 1 ismerely an example, and the functional configuration of the informationprocessing device 30 according to the present embodiment is not limitedto such an example. The functional configuration of the informationprocessing device 30 according to the present embodiment can be flexiblymodified in accordance with specifications and applications.

[Processing Procedure of Information Processing Device According toFirst Embodiment]

Next, an example of a processing procedure of the information processingdevice 30 according to the first embodiment will be described. FIG. 3 isa flowchart illustrating an example of a processing procedure executedby the information processing device 30 according to the firstembodiment. The processing procedure illustrated in FIG. 3 is actualizedby execution of a program by the control unit 32 of the informationprocessing device 30. The processing procedure illustrated in FIG. 3 isrepeatedly executed by the control unit 32.

As illustrated in FIG. 3, the control unit 32 of the informationprocessing device 30 executes a process of measuring a real physicalbody (step S10). For example, the control unit 32 measures a geometricshape in the real environment P as a real physical body based on thesensor information of the sensor unit 10, and stores measurementinformation indicating a measurement result in the storage unit 31. Byexecuting the process of step S10, the control unit 32 functions as themeasurement unit 321 described above. After completion of the process ofstep S10, the control unit 32 proceeds to the process of step S20.

The control unit 32 executes a process of recognizing a physical body(step S20). For example, the control unit 32 recognizes a physical bodyin the real environment P based on the measurement information and thephysical body recognition model 311. The control unit 32 recognizes astructure, a category, and the like for each of the recognized physicalbodies.

An example of the process of recognizing a physical body by the controlunit 32 will be described with reference to FIGS. 4 and 5. FIG. 4 is adiagram illustrating an example in which the information processingdevice 30 recognizes a physical body. FIG. 5 is a diagram illustratingan example of the physical body recognition model 311.

In the example illustrated in FIG. 4, the control unit 32 searches for amodel that matches or resembles the geometric shape indicated by themeasurement information from among the models of the physical bodyrecognition model 311, and recognizes that a physical body object R is asofa. In addition, an example illustrated in FIG. 5 represents arelationship between a model 311M of the physical body recognition model311 and shape information 311A. The shape information 311A is stored inthe physical body recognition model 311. The shape information 311Aincludes information of a vertex definition and a mesh definition. Thevertex definition defines vertex coordinates XYZ of points v1, v2, v3,v4, v5, and the like, vertex colors RGB, and a structural label PLregarding the model 311M. The structural label PL is an element of a setSG. In an example illustrated in FIG. 5, the set SG is a set regarding asofa, and includes a seat, a backrest, a support, legs, and joints. Themesh definition defines an index list of three vertices constituting atriangle in the model 311M. The control unit 32 compares the shapeinformation 311A of the physical body recognition model 311 with themeasured geometric shape, and searches for the model 311M from thephysical body recognition model 311.

Returning to FIG. 4, the control unit 32 recognizes the structureindicated by the model 311M searched from the physical body recognitionmodel 311 as a structure of the physical body object R. In this case,the control unit 32 recognizes that the physical body object R includeselements such as a seat R1, a backrest R2, a support R3, legs R4, andjoints R5.

Returning to FIG. 3, after completion of the process of step S20, thecontrol unit 32 proceeds to the process of step S30. Note that thecontrol unit 32 executes the process of step S20, thereby functioning asthe first recognition unit 322 described above.

The control unit 32 executes a process of recognizing the structure andphysical properties (step S30). For example, the control unit 32searches for a model that matches or resembles the recognized physicalbody from among the models of the structure/physical property model 312,and recognizes the structure and physical properties indicated by themodel as the structure and physical properties of the physical body.

The structure/physical property model 312 stores physical propertyinformation 312A illustrated in FIG. 6 in association with the model311M. FIG. 6 is a diagram illustrating an example of thestructure/physical property model 312. The physical property information312A indicates a relationship between an element of the model 311M and aphysical property. For example, in the physical property information312A, information such as the number of parts, mass, rigidity, softness,load capacity, thermal conductivity, and material is set for each ofelements of the model 311M. Note that the higher the rigidity of thephysical property information 312A, the less likely the physical body isto be deformed. The higher the softness of the physical propertyinformation 312A, the softer the physical body is.

The control unit 32 extracts the physical property information 312Aassociated with the model 311M from the structure/physical propertymodel 312, and recognizes the information as physical properties of theelements of the physical body object R based on the physical propertyinformation 312A. For example, regarding the physical body object R ofthe sofa illustrated in FIG. 4, the control unit 32 recognizes, from theextracted physical property information 312A, that the physical bodyobject R has physical properties such as high softness in the seat R1,moderate softness in the backrest R2, and high rigidity in the supportR3.

Returning to FIG. 3, after associating the recognition result of thephysical body object R with the measurement information, the controlunit 32 proceeds to the process of step S40. Note that the control unit32 executes the process of step S30, thereby functioning as the secondrecognition unit 323 described above.

The control unit 32 executes a process of detecting a missing defect(step S40). For example, the control unit 32 detects a structuralmissing defect of the recognized physical body object R based on thestructural condition of the physical body provided in the structuralcondition DB 313. The control unit 32 executes the process of step S40,thereby functioning as the missing defect detection unit 324 describedabove.

FIG. 7 is a flowchart illustrating an example of a processing procedureof missing defect detection executed by the information processingdevice 30 according to the first embodiment. The processing procedureillustrated in FIG. 7 is implemented by execution of the process of stepS40 by the control unit 32. As illustrated in FIG. 7, the control unit32 acquires measurement information (step S401). For example, thecontrol unit 32 acquires measurement information associated with arecognition result. The control unit 32 acquires the structuralcondition (step S402). For example, the control unit 32 acquires, fromthe structural condition DB 313, a structural condition associated witha model that matches or resembles the recognized physical body object R.

After completion of the processes in steps S401 and S402, the controlunit 32 detects a structural missing defect (step S403). For example,the control unit 32 compares the measurement information with thestructural condition, and detects absence of an essential part of thephysical body object R. Specifically, in a case where the physical bodyobject R is a sofa, the control unit 32 detects a missing defect of thephysical body object R based on structural and positional relationshipsof parts, such as insufficient number of legs, and the absence of legsunder the seat. The control unit 32 stores the detection result in thestorage unit 31 and proceeds to the process of step S404.

The control unit 32 determines whether there is a missing defect basedon the detection result (step S404). In a case where it is determinedthat there is a missing defect (Yes in step S404), the control unit 32finishes the processing procedure illustrated in FIG. 7 and proceeds toa process of complementing the missing defect in step S41 illustrated inFIG. 3 (step S408). The process of step S41 will be described below.

When having determined that there is no missing defect (No in stepS404), the control unit 32 proceeds to the process of step S405. Thecontrol unit 32 evaluates safety by physical simulation (step S405). Forexample, by executing the physical simulation described above, thecontrol unit 32 evaluates the missing defect and the safety of thephysical body object R. For example, when the physical body object R isfound to lack safety by execution of the physical simulation, thecontrol unit 32 recognizes that there is a missing defect in thephysical body object R. When the physical body object R is found to havesafety, the control unit 32 recognizes that there is no missing defectin the physical body object R.

FIG. 8 is a diagram illustrating an example in which the informationprocessing device 30 detects a missing defect in a physical body. Asillustrated in FIG. 8, the information processing device 30 measures aphysical body object R′, and recognizes that the physical body object R′includes a seat R1, a backrest R2, a support R3, and legs R4. In thiscase, the control unit 32 detects a missing defect in which the rightside of the physical body object R′ is missing based on the positionalrelationship between the seat R1, the backrest R2, the support R3, andthe legs R4 of the physical body object R′. For example, when the numberof legs is insufficient, the sofa falls or tilts due to the action ofgravity. For example, the shape of the sofa cannot be maintained anddeforms unless a highly soft portion is supported by a high rigiditypart. The physical simulation is a simulation for evaluating thestability of the recognized physical body. In the example illustrated inFIG. 8, in a case where evaluation is performed to apply force to thephysical body object R′ in a plurality of directions by physicalsimulation, the control unit 32 evaluates that the stability is low(poor) because the physical body object R′ is inclined in a direction ofan arrow F.

Returning to FIG. 7, the control unit 32 determines whether there is amissing defect based on the evaluation result of step S405 (step S406).When having determined that there is no missing defect (No in stepS406), the control unit 32 finishes the processing procedure illustratedin FIG. 7 and proceeds to the process of estimating the interaction instep S50 illustrated in FIG. 3 (step S407). The process of step S50 willbe described below.

When having determined that there is a missing defect (Yes in stepS406), the control unit 32 finishes the processing procedure illustratedin FIG. 7 and proceeds to a process of complementing the missing defectin step S41 illustrated in FIG. 3 (step S408).

Returning to FIG. 3, the control unit 32 executes the process ofcomplementing the missing defect (step S41). For example, the controlunit 32 recognizes a missing defect portion of the physical body objectR based on data such as the shape, structure, and physical properties ofthe physical body included in the 3D model DB 314, and complements themissing defect portion. After complementing the missing defect, thecontrol unit 32 adds information corresponding to the complementedportion to the first information 31A. The control unit 32 executes theprocess of step S41, thereby functioning as the missing defectcomplementing unit 324A described above.

FIG. 9 is a diagram illustrating an example in which the informationprocessing device 30 complements a missing defect in a physical body. Asillustrated in FIG. 9, the information processing device 30 measures aphysical body object R′, and recognizes that the physical body object R′includes a seat R1, a backrest R2, a support R3, and legs R4. In thiscase, the control unit 32 extracts a 3D model that matches or resemblesthe physical body object R′ from the 3D model DB 314. The control unit32 recognizes a missing defect portion of the physical body object R′based on data such as the shape, structure, and physical properties ofthe 3D model, and complements the missing defect portion. In the exampleillustrated in FIG. 9, a portion on the right side of the physical bodyobject R′ is missing, and thus, the control unit 32 complements theportion on the right side of the physical body object R′ based on the 3Dmodel. As a result, the control unit 32 can obtain a physical bodyobject RC including a seat R1, a backrest R2, a support R3, legs R4, andjoints R5.

Returning to FIG. 3, after adding the information corresponding to theportion obtained by complementing the missing defect to the firstinformation 31A, the control unit 32 proceeds to the process of stepS50.

The control unit 32 executes a process of estimating interaction (stepS50). For example, the control unit 32 specifies the positionalrelationship between the recognized physical bodies and estimates aninteraction between the physical bodies based on the first information31A for each of the physical body objects R. Note that the control unit32 executes the process of step S50, thereby functioning as theestimation unit 325 described above.

FIG. 10 is a flowchart illustrating an example of a processing procedureof interaction estimation executed by the information processing device30 according to the first embodiment. The processing procedureillustrated in FIG. 10 is implemented by execution of the process ofstep S50 by the control unit 32. As illustrated in FIG. 10, the controlunit 32 acquires measurement information (step S501). For example, thecontrol unit 32 acquires measurement information associated with arecognition result. When having acquired the measurement result, thecontrol unit 32 proceeds to the process of step S502.

The control unit 32 estimates an interaction between real physicalbodies (step S502). For example, the control unit 32 specifies thepositional relationship between the recognized physical bodies andstores, in the storage unit 31, a result of estimating the interactionbetween the physical bodies based on the first information 31A for eachof the physical body objects R. For example, in a case where there is nooccurrence of interaction between the physical bodies, the control unit32 stores a result indicating the impossibility of estimation ofinteraction in the storage unit 31. After storing the estimation resultin the storage unit 31, the control unit 32 proceeds to the process ofstep S503.

The control unit 32 determines whether there is an interaction based onthe estimation result of step S502 (step S503). When having determinedthat there is no interaction (No in step S503), the control unit 32proceeds to a process of specifying an interaction in step S60illustrated in FIG. 3 to be described below (step S504). When havingdetermined that there is an interaction (Yes in step S503), the controlunit 32 proceeds to the process of step S505.

The control unit 32 evaluates the interaction between real physicalbodies (step S505). For example, using a result of simulation, machinelearning, and the like, the control unit 32 evaluates the interactionbased on the degree of deformation of the physical body due to thepresence or absence of the interaction. For example, the control unit 32evaluates the interaction based on the degree of deformation of thephysical body by using the shape of the part having no interaction in anidentical physical body. Furthermore, the control unit 32 may acquire asimilar 3D model from the 3D model DB 314 and evaluate the interactionusing the acquired 3D model.

FIG. 11 is a diagram illustrating an example in which the informationprocessing device 30 evaluates an interaction between physical bodies.In a scene SN1 illustrated in FIG. 11, based on a measurement result,the control unit 32 recognizes a physical body object RA being a sofaand a physical body object RB being a human sitting on the physical bodyobject RA. Subsequently, in a scene SN2, the control unit 32 recognizesa seat R1 of the physical body object RA by a process of recognizing thephysical body. In a scene SN3, the control unit 32 evaluates thephysical properties of the seat R1 of the physical body object RA basedon a deformation amount E of the portion of the seat R1 of the physicalbody object RA on which the physical body object RA is sitting and themass of the physical body object RB. Note that the mass of the physicalbody object RB includes, for example, the mass predicted from the size,the mass (weight) of the recognized individual, and the like. Thecontrol unit 32 stores the evaluation result in the storage unit 31.

Returning to FIG. 10, when having stored the evaluation result in thestorage unit 31, the control unit 32 proceeds to a process of correctingthe physical properties in step S51 illustrated in FIG. 3 (step S506).

Returning to FIG. 3, the control unit 32 executes the process ofcorrecting physical properties (step S51). For example, there is apossibility that the recognition result of the physical body includes anerror. Therefore, the control unit 32 corrects the physical propertiesso as to approach the deformation amount obtained in the evaluation ofthe interaction between the real physical bodies, for example. Aftercorrecting the first information 31A of the physical body based on theevaluation result, the control unit 32 proceeds to the process of stepS60. The control unit 32 executes the process of step S51, therebyfunctioning as the correction unit 325A described above.

The control unit 32 executes a process of specifying an interaction(step S60). For example, the control unit 32 specifies an arrangement ofthe object C in the virtual space V so that the arrangement is capableof expressing the interaction with the physical body object based on thearrangement condition 31C of the object C and on the first information31A. Note that the control unit 32 executes the process of step S60,thereby functioning as the specifying unit 326 described above.

FIG. 12 is a flowchart illustrating an example of a processing procedureof specifying an interaction executed by the information processingdevice 30 according to the first embodiment. The processing procedureillustrated in FIG. 12 is implemented by execution of the process ofstep S60 by the control unit 32. As illustrated in FIG. 12, the controlunit 32 acquires the first information 31A (step S601). For example, thecontrol unit 32 acquires the first information 31A of the physical bodyobject R being recognized. The control unit 32 acquires the secondinformation 31B from the object DB 315 (step S602). For example, thecontrol unit 32 acquires, from the object DB 315, the second information31B including information regarding the structure and physicalproperties of the object C to be displayed. The control unit 32 acquiresthe arrangement condition 31C of the object C (step S603). For example,the control unit 32 acquires the arrangement condition 31C forexpressing the interaction of the object C from the interaction DB 316.After completion of the process of step S603, the control unit 32proceeds to the process of step S604.

Although the processing procedure illustrated in FIG. 12 illustrates aprocedure of executing the processing in the order of steps S601, S602,and S603, the processing procedure illustrated in FIG. 12 is not limitedto this order. The processing procedure in FIG. 12 may be performed inan order changed from the order of steps S601, S602, and S603, may besimultaneously performed, or may be performed as one process.

The control unit 32 selects a part applicable for interaction (stepS604). For example, the control unit 32 selects a part that can expressthe interaction between the physical body and the object C based on theacquired arrangement condition 31C. For example, in a case where thearrangement condition 31C is “sitting down”, the control unit 32 selectsa part of a physical body on which the object C can sit down, andselects a part of the object C in a case where the object C sits down.After completion of the process of step S604, the control unit 32proceeds to the process of step S605.

The control unit 32 specifies an arrangement of the object C in thevirtual space V (step S605). For example, the control unit 32 specifiesan optimum arrangement of the object C in the virtual space V so as tosatisfy the arrangement condition 31C based on the selected part of thephysical body and the part of the object C. For example, the controlunit 32 specifies the arrangement of the object C based on a physicalconstraint between the physical body and the object C. The physicalconstraint includes constraints such as physical contact, collision, andload capacity of the physical body, for example. For example, thecontrol unit 32 specifies an arrangement that satisfies the arrangementcondition 31C based on the positional relationship between the physicalbody and the object C in the virtual space V. For example, the controlunit 32 specifies an arrangement that is capable of expressing thecharacteristic (personality) of the object C and that satisfies thearrangement condition 31C. After having stored the specified result inthe storage unit 31, the control unit 32 proceeds to the process of stepS606.

The control unit 32 determines whether the arrangement is feasible basedon the specifying result of step S605 (step S606). For example, in acase where the specifying result indicates that the arrangement has beensuccessfully specified, the control unit 32 determines that thearrangement is feasible. When determining that the arrangement isfeasible (Yes in step S606), the control unit 32 proceeds to the processof reflecting the interaction in step S70 illustrated in FIG. 3 to bedescribed below (step S607). When having determined that the arrangementis not feasible (No in step S606), the control unit 32 proceeds to theprocess of step S608.

The control unit 32 determines whether there is an alternativearrangement condition 31C (step S608). For example, in a case whereanother arrangement condition 31C of the object C exists in theinteraction DB 316, the control unit 32 determines that there is analternative arrangement condition 31C. When having determined that thereis an alternative arrangement condition 31C (Yes in step S608), thecontrol unit 32 proceeds to the process of step S609. The control unit32 acquires the alternative arrangement condition 31C (step S609). Whenhaving acquired the alternative arrangement condition 31C, the controlunit 32 returns to the process of step S604 already described, andcontinues the processes of step S604 and subsequent steps. That is, thecontrol unit 32 executes the processes from step S604 to step S606related to the alternative arrangement condition 31C.

When having determined that there is no alternative arrangementcondition 31C (No in step S608), the control unit 32 cannot arrange theobject C in the virtual space V, and thus stops the process (step S610).

FIG. 13 is a diagram illustrating an example in which the informationprocessing device 30 specifies an arrangement of a physical body. Asillustrated in FIG. 13, the control unit 32 recognizes a physical bodyobject R being a sofa. The control unit 32 recognizes that thearrangement condition 31C for the object C is “sitting down”. In thiscase, the control unit 32 selects the seat R1 of the physical bodyobject R as a part PT1 of the physical body object R, and selects thebuttocks of the object C in the sitting posture as a part PT2. When thepart PT2 of the object C can be arranged at the part PT1 of the physicalbody object R, the control unit 32 specifies the position in the virtualspace V corresponding to the part PT1 of the physical body object R asthe arrangement of the object C.

Furthermore, in a case where the part PT2 of the object C cannot bearranged at the part PT1 of the physical body object R due to thepositional relationship of the physical body object R with anotherphysical body, the control unit 32 acquires an alternative arrangementcondition 31C. For example, here, the alternative arrangement condition31C is “lying”. In this case, the control unit 32 selects the seat R1 ofthe physical body object R as a part PT1 of the physical body object R,and selects the abdomen, legs, and the like of the object C in the lyingposture as a part. When the part of the object C can be arranged at thepart PT1 of the physical body object R, the control unit 32 specifiesthe position in the virtual space V corresponding to the part PT1 of thephysical body object R as the arrangement of the object C.

Returning to FIG. 3, when the process of step S60 is completed, thecontrol unit 32 executes a process of reflecting the interaction (stepS70). For example, the control unit 32 executes a process of expressingan interaction between the physical body object R and the object C basedon the specified arrangement of the object C in the virtual space V.Note that the control unit 32 executes the process of step S70, therebyfunctioning as the processing unit 327 described above.

FIG. 14 is a diagram illustrating an example in which the informationprocessing device 30 expresses an interaction between the physical bodyobject R and the object C. As illustrated in FIG. 14, the control unit32 arranges the object C such that the part PT2 of the object C comes incontact with the part PT1 of the seat R1 of the physical body object R.In this case, the control unit 32 determines an expression of sinking asan interaction in a portion of the part PT1 of the physical body objectR with which the part PT2 comes in contact. For example, the controlunit 32 calculates the amount of sink in the part PT2 of the physicalbody object R based on the physical properties of the physical bodyobject R and the weight of the object C. As a result, the control unit32 stores, in the storage unit 31, a processing result includingposition information indicating the position of individual parts of theobject C in the virtual space V and deformation information indicatingthe amount of deformation and the like in the part PT2 of the physicalbody object R.

Returning to FIG. 3, the control unit 32 executes a process ofcontrolling display on the display device 20 (step S80). For example,the control unit 32 controls to create a VR image based on theprocessing result and the map information 31M and display the created VRimage on the display device 20. As a result, the display device 20displays a VR image expressing the interaction between the physical bodyobject R obtained by capturing the real environment into the virtualspace, and the object C. Note that the control unit 32 executes theprocess of step S80, thereby functioning as the display control unit 328described above. After completion of the process of step S80, thecontrol unit 32 finishes the processing procedure illustrated in FIG. 3.

As described above, after capturing a real physical body into thevirtual space V as the physical body object R, the informationprocessing device 30 according to the first embodiment recognizes thefirst information 31A indicating the structure and physical propertiesof the physical body object R and stores the first information 31A inthe storage unit 31. The information processing device 30 specifies anarrangement of the object C representing a virtual object in the virtualspace V so that the arrangement is capable of expressing the interactionwith the physical body object R based on the arrangement condition 31Cof the object C and on the first information 31A.

For example, as illustrated in FIG. 14, in a case where the physicalbody object R is a sofa and the arrangement condition 31C of the objectC is “sitting down”, the information processing device 30 specifies theseat portion of the sofa in the virtual space V as a position where theobject C can be arranged. With this configuration, by arranging theobject C at a position where the object C can interact with the physicalbody object R in the virtual space V into which the real environment hasbeen captured, the information processing device 30 can suppress a gapoccurring between the physical body object R and the object C in thevirtual space V. As a result, the information processing device 30 cansuppress the strangeness or incompatibility of the object C displayed inthe virtual space V into which the measured real environment has beencaptured.

Furthermore, the information processing device 30 specifies anarrangement of the object C in the virtual space V so that thearrangement satisfies physical conditions between the part of the objectC and the physical body object R corresponding to the arrangementcondition 31C and so that the arrangement is capable of expressing theinteraction with the physical body object R. With this configuration,the information processing device 30 specifies an arrangement of theobject C in the virtual space V so that the arrangement satisfies thephysical condition between the part of the object C and the physicalbody object R, making it possible to arrange the object C at anappropriate position in the virtual space where the interaction can beexpressed. As a result, the information processing device 30 can achievenatural expression of the interaction between the physical body object Rand the object C.

Second Embodiment

[Outline of Display System According to Second Embodiment]

FIG. 15 is a diagram illustrating an example of a display systemaccording to a second embodiment. A display system 100 illustrated inFIG. 15 includes a sensor unit 10, a display device 20, and aninformation processing device 30, similarly to the display system 100 ofthe first embodiment. Note that description of the configuration similarto the display system 100 according to the first embodiment will beomitted. The display system 100 is mounted on the head of a user U, anddisplays, on the display device 20, an image of a virtual space V intowhich a measured real environment P has been captured, under the controlof the information processing device 30. With this configuration, theuser U recognizes the virtual space V displayed in front of eyes EY.

The display system 100 is capable of wirelessly communicating with anoperation input unit 50. The operation input unit 50 has a function ofinputting an operation of the user U, for example. The operation inputunit 50 includes, for example, input devices such as a controller of agame machine, a hardware button, and a touch panel. The operation inputunit 50 transmits information indicating the operation result of theuser U to the display system 100. Note that the operation input unit 50may transmit information to the display system 100 via a game machine,for example. The operation input unit 50 may be formed integrally withthe display system 100.

The information processing device 30 includes a storage unit 31 and acontrol unit 32. The control unit 32 includes functional units such as ameasurement unit 321, a first recognition unit 322, a second recognitionunit 323, a missing defect detection unit 324, an estimation unit 325, aspecifying unit 326, a processing unit 327, and a display control unit328.

In an example illustrated in FIG. 15, the information processing device30 provides a function of capturing a real environment P in a kitcheninto the virtual space V and virtually cooking in accordance with theoperation of the operation input unit 50 by the user U. For example, theinformation processing device 30 arranges the object C in the virtualspace V, and displays an image in which the object C performs cookingusing a physical body object R10 on the display device 20. The physicalbody object R10 includes a physical body used for cooking, such as akitchen knife, a smaller knife, a cutting board, a sink, a cookingstove, a frying pan, and a pot, for example,

[Processing Procedure of Information Processing Device According toSecond Embodiment]

Next, an example of a processing procedure of the information processingdevice 30 according to the second embodiment will be described. FIG. 16is a flowchart illustrating an example of a processing procedureexecuted by the information processing device 30 according to the secondembodiment. The processing procedure illustrated in FIG. 16 isactualized by execution of a program by the control unit 32 of theinformation processing device 30. The processing procedure illustratedin FIG. 16 is repeatedly executed by the control unit 32.

As illustrated in FIG. 16, the control unit 32 of the informationprocessing device 30 executes a process of measuring a real physicalbody (step S10). For example, the control unit 32 stores measurementinformation indicating a real physical body in the real environment P inthe storage unit 31 based on sensor information of the sensor unit 10.After completion of the process of step S10, the control unit 32proceeds to the process of step S20.

The control unit 32 executes a process of recognizing a physical body(step S20). For example, the control unit 32 recognizes a physical bodyto be used in cooking in the real environment P based on the measurementinformation and the physical body recognition model 311. Aftercompletion of the process of step S20, the control unit 32 proceeds tothe process of step S30.

The control unit 32 executes a process of recognizing the structure andphysical properties (step S30). For example, the control unit 32searches for a model that matches or resembles the recognized physicalbody from among the models of the structure/physical property model 312,and recognizes the structure and physical properties indicated by themodel as the structure and physical properties of the physical body. Forexample, when having recognized that the physical body object R10 is akitchen knife, the control unit 32 recognizes that the physical bodyobject R10 has a blade and a handle. For example, the control unit 32recognizes that the blade of the physical body object R10 has a cuttingability and the handle has high rigidity. After completion of theprocess of step S30, the control unit 32 proceeds to the process of stepS40.

The control unit 32 executes a process of detecting a missing defect(step S40). In a case where there is a missing defect, the control unit32 executes the process of complementing the missing defect in step S41and then proceeds to the process of step S50. In addition, in a casewhere there is no missing defect, the control unit 32 proceeds to theprocess of step S50.

The control unit 32 executes a process of estimating interaction (stepS50). For example, the control unit 32 specifies the positionalrelationship between the recognized physical bodies in the kitchen andestimates the interaction between the physical bodies based on the firstinformation 31A for each of the physical body objects R. In a case wherethere is interaction, the control unit 32 executes the process ofcorrecting the physical properties in step S51 and proceeds to theprocess of step S61. Furthermore, in a case where there is nointeraction, the control unit 32 proceeds to the process of step S61.

The control unit 32 executes a process of specifying interaction withthe user (step S61). For example, the control unit 32 specifies anarrangement of the object C in the virtual space V so that thearrangement is capable of expressing the interaction with the physicalbody object based on the arrangement condition 31C of the object Coperated by the user and on the first information 31A. For example, in acase where the arrangement condition 31C is “applying a kitchen knife toan ingredient”, the control unit 32 specifies an arrangement in thevirtual space V so that the arrangement is capable of expressing aninteraction between two physical body objects R, namely, the ingredientand the kitchen knife, and the object C. After completion of the processof step S60, the control unit 32 proceeds to the process of step S70.

The control unit 32 executes a process of reflecting an interaction(step S70). For example, the control unit 32 executes a process ofexpressing an interaction between the physical body object R and theobject C based on the specified arrangement of the object C in thevirtual space V. For example, the control unit 32 executes a process ofexpressing an interaction occurring among the plurality of physical bodyobjects R in the virtual space V based on a plurality of pieces of firstinformation 31A and the motion of the object C. For example, in a casewhere the arrangement condition 31C is “applying a kitchen knife to aningredient”, the control unit 32 executes a process of expressingdeformation and cutting of the ingredient by the interaction between theingredient and the kitchen knife by applying the kitchen knife to theingredient. Furthermore, the control unit 32 may execute a process ofproviding the user U with the weight of the physical body object R by avibration function or the like of the operation input unit 50 as theinteraction between the physical body object R and the object C.Furthermore, in a case where the object C touches a dangerous part ofthe physical body object R, the control unit 32 may execute a process ofnotifying the user U of danger as the interaction between the physicalbody object R and the object C. After completion of the process of stepS70, the control unit 32 proceeds to the process of step S80.

The control unit 32 executes a process of controlling display on thedisplay device 20 (step S80). For example, the control unit 32 controlsto create a VR image based on the processing result and the mapinformation 31M and display the created VR image on the display device20. As a result, the display device 20 displays a VR image expressingthe interaction in which the object C performs cooking using thephysical body objects R obtained by capturing the real environment intothe virtual space. After completion of the process of step S80, thecontrol unit 32 finishes the processing procedure illustrated in FIG.16.

As described above, the information processing device 30 according tothe second embodiment executes the process of expressing the interactionoccurring between the plurality of physical body objects R in thevirtual space V based on the plurality of pieces of first information31A and the motion of the object C.

For example, in the virtual space V illustrated in FIG. 15, it isassumed that the physical body object R10 is a kitchen knife, and thearrangement condition 31C of the object C is “applying a kitchen knifeto an ingredient”. In this case, the information processing device 30expresses an interaction in which the kitchen knife cuts the ingredientin the virtual space V based on the first information 31A regarding theingredient and the motion of the object C in the virtual space V. As aresult, by recognizing tools, ingredients, facilities, and the like inthe real environment as physical bodies, the information processingdevice 30 can provide the user with virtual cooking in the virtual spaceV, leading to an effective use of the live-action VR.

The above-described second embodiment is an example, and variousmodifications and applications are possible. The information processingdevice 30 of the second embodiment may be applied to other embodimentsand the like.

Third Embodiment

[Outline of Display System According to Third Embodiment]

FIG. 17 is a diagram illustrating an example of a display systemaccording to a third embodiment. Similarly to the display system 100 ofthe first embodiment, a display system 100 illustrated in FIG. 17includes a sensor unit 10, a display device 20, and an informationprocessing device 30. Note that description of the configuration similarto the display system 100 according to the first and second embodimentswill be omitted. The display system 100 is mounted on the head of a userU, and displays, on the display device 20, an image of a virtual space Vinto which a measured real environment P has been captured, under thecontrol of the information processing device 30. With thisconfiguration, the user U recognizes the virtual space V displayed infront of eyes EY.

The information processing device 30 includes a storage unit 31 and acontrol unit 32. The control unit 32 includes functional units such as ameasurement unit 321, a first recognition unit 322, a second recognitionunit 323, a missing defect detection unit 324, an estimation unit 325, aspecifying unit 326, a processing unit 327, and a display control unit328.

In an example illustrated in FIG. 17, the information processing device30 provides a function of capturing a real environment P including afishing rod P1 and a pond into the virtual space V and virtually fishingin accordance with the operation of the operation input unit 50 by theuser U. For example, the information processing device 30 arranges theobject C in the virtual space V, and displays an image in which theobject C performs fishing using a physical body object R on the displaydevice 20. The physical body object R is a fishing rod.

[Processing Procedure of Information Processing Device According toThird Embodiment]

Next, an example of a processing procedure of the information processingdevice 30 according to the third embodiment will be described. FIG. 18is a flowchart illustrating an example of a processing procedureexecuted by the information processing device 30 according to the thirdembodiment. The processing procedure illustrated in FIG. 18 isactualized by execution of a program by the control unit 32 of theinformation processing device 30 The processing procedure illustrated inFIG. 18 is repeatedly executed by the control unit 32.

As illustrated in FIG. 18, the control unit 32 of the informationprocessing device 30 executes a process of measuring a real physicalbody (step S10). For example, the control unit 32 stores measurementinformation indicating a real physical body in the real environment P inthe storage unit 31 based on sensor information of the sensor unit 10.The control unit 32 may simultaneously measure or separately measure thereal environment P and the fishing rod P1 being a physical body. Aftercompletion of the process of step S10, the control unit 32 proceeds tothe process of step S20.

The control unit 32 executes a process of recognizing a physical body(step S20). For example, the control unit 32 recognizes the pond and thefishing rod P1 being the real environment P based on the measurementinformation and the physical body recognition model 311. Aftercompletion of the process of step S20, the control unit 32 proceeds tothe process of step S30.

The control unit 32 executes a process of recognizing the structure andphysical properties (step S30). For example, the control unit 32searches for a model that matches or resembles the recognized physicalbody from among the models of the structure/physical property model 312,and recognizes the structure and physical properties indicated by themodel as the structure and physical properties of the physical body. Forexample, in an example illustrated in FIG. 17, when having recognizedthat a physical body object R0 in the real environment P is a pond, thecontrol unit 32 recognizes that the physical property of the physicalbody object R0 is water. For example, when having recognized that thephysical body object R is a fishing rod, the control unit 32 recognizesthat the physical body object R has a tip R11 and a rod R12. Forexample, the control unit 32 recognizes a physical property that the tipR11 has high flexibility and the rod R12 has high rigidity in thephysical body object R. Returning to FIG. 18, when having completed theprocess of step S30, the control unit 32 proceeds to the process of stepS40.

The control unit 32 executes a process of detecting a missing defect(step S40). In a case where there is a missing defect, the control unit32 executes a process of complementing the missing defect in step S41and then proceeds to the process of step S50. In addition, in a casewhere there is no missing defect, the control unit 32 proceeds to theprocess of step S50.

The control unit 32 executes a process of estimating interaction (stepS50). For example, the control unit 32 specifies the positionalrelationship between the recognized physical properties, and estimatesthe interaction between the physical bodies based on the firstinformation 31A for each of the physical body objects R and R0. In acase where there is interaction, the control unit 32 executes a processof correcting the physical properties in step S51, and then proceeds tothe process of steps S60 and S62. In addition, in a case where there isno interaction, the control unit 32 proceeds to the process of steps S60and S62.

The control unit 32 executes a process of specifying an interaction(step S60). For example, the control unit 32 specifies an arrangement ofthe object C in the virtual space V so that the arrangement is capableof expressing the interaction with the physical body object based on thearrangement condition 31C of the object C and on the first information31A. For example, when the arrangement condition 31C is “fishing”, thecontrol unit 32 specifies the arrangement of the object C in the virtualspace V so that the object C is located at an edge of a pond or thelike.

Furthermore, the control unit 32 specifies an arrangement of an objectC2 in the virtual space V based on the arrangement condition 31C of theobject C2 operable by the object C and on the first information 31A. Theobject C2 is an example of a third object. The storage unit 31 storesthe arrangement condition 31C of the third object. For example, in acase where the third object is “fish”, the arrangement condition 31Cindicates a condition for arrangement in water. In the virtual space Villustrated in FIG. 17, the control unit 32 specifies the arrangement ofthe object C in the virtual space V such that the object C2 being a fishis positioned in a pond. After completion of the process of step S60,the control unit 32 proceeds to the process of step S70.

The control unit 32 executes a process of reflecting an interaction(step S70). For example, the control unit 32 executes a process ofexpressing an interaction between the physical body object R and theobject C based on the specified arrangement of the object C in thevirtual space V. For example, the control unit 32 executes processes ofexpressing the interaction when arranging the object C to stand on anedge of the pond in the virtual space V and arranging the object C2 inthe pond. Since the object C2 is fish, the control unit 32 executesprocesses of expressing an interaction represented by the movement ofthe object C2 toward a bait after the bait is thrown into the pond, orthe movement of the object C2 being caught when the user lifts thefishing rod at an appropriate timing, for example. After completion ofthe process of step S70, the control unit 32 proceeds to the process ofstep S80.

As another flow, the control unit 32 executes a process of specifyingthe interaction of the user U (step S62). For example, the control unit32 specifies the interaction among the physical body object R, theobject C, and the object C2 based on the first information 31A of thephysical body object R and the position information of the object C2.For example, the control unit 32 specifies an interaction related to themass of the fishing rod to be fed back to the user U. Note that thecontrol unit 32 executes the process of step S62, thereby functioning asthe specifying unit 326 described above. After completion of the processof step S62, the control unit 32 proceeds to the process of step S72.

The control unit 32 executes a process of reflecting the interaction ofthe user U (step S72). For example, the control unit 32 executes aprocess of expressing an interaction between the physical body object Rand another object in the specified virtual space V. For example, thecontrol unit 32 executes a process of providing the user U with theweight of the physical body object R by the vibration function or thelike of the operation input unit 50 as the interaction between thephysical body object R and the other object. Note that the control unit32 executes the process of step S72, thereby functioning as theprocessing unit 327 described above. After completion of the process ofstep S72, the control unit 32 proceeds to the process of step S80.

The control unit 32 executes a process of controlling display on thedisplay device 20 (step S80). For example, the control unit 32 controlsto create a VR image based on the processing result and the mapinformation 31M and display the created VR image on the display device20. As a result, the display device 20 displays a VR image expressingthe interaction in which the object C performs fishing using thephysical body object R obtained by capturing the real environment intothe virtual space. After completion of the process of step S80, thecontrol unit 32 finishes the processing procedure illustrated in FIG.18.

As described above, the information processing device 30 according tothe third embodiment specifies arrangement of the object C2 in thevirtual space V based on the arrangement condition 31C of the object C2operable by the object C and on the first information 31A. Theinformation processing device 30 executes the process of expressing aninteraction occurring between the object C2 and the physical body objectR based on a characteristic of the object C2 and on the firstinformation 31A.

For example, in the virtual space V illustrated in FIG. 17, it isassumed that the physical body object R is a fishing rod and thearrangement condition 31C of the object C is “fishing”. In this case,the information processing device 30 expresses an interaction in thecase of fishing in the virtual space V based on the object C2 indicatinga fish in the virtual space V and the first information 31A regardingthe fishing rod. As a result, by recognizing the fishing rod in the realenvironment as a physical body, the information processing device 30 canprovide the user with virtual fishing in the virtual space V, leading toan effective use of the live-action VR.

The above-described third embodiment is an example, and variousmodifications and applications are possible. The information processingdevice 30 of the third embodiment may be applied to other embodimentsand the like.

[Hardware Configuration]

The information processing device 30 according to the present embodimentdescribed above may be actualized by a computer 1000 having aconfiguration as illustrated in FIG. 19, for example. Hereinafter, theinformation processing device 30 according to an embodiment will bedescribed as an example. FIG. 19 is a hardware configuration diagramillustrating an example of the computer 1000 that actualizes functionsof the information processing device 30. The computer 1000 includes aCPU 1100, a RAM 1200, a read only memory (ROM) 1300, a hard disk drive(HDD) 1400, a communication interface 1500, and an input/outputinterface 1600. Individual components of the computer 1000 areinterconnected by a bus 1050.

The CPU 1100 operates based on a program stored in the ROM 1300 or theHDD 1400 so as to control each of components. For example, the CPU 1100develops a program stored in the ROM 1300 or the HDD 1400 into the RAM1200 and executes processes corresponding to various programs.

The ROM 1300 stores a boot program such as a basic input output system(BIOS) executed by the CPU 1100 when the computer 1000 starts up, aprogram dependent on hardware of the computer 1000, or the like.

The HDD 1400 is a non-transitory computer-readable recording medium thatrecords a program executed by the CPU 1100, data used by the program, orthe like. Specifically, the HDD 1400 is a recording medium that recordsan information processing program according to the present disclosure,which is an example of program data 1450.

The communication interface 1500 is an interface for connecting thecomputer 1000 to an external network 1550 (for example, the Internet).For example, the CPU 1100 receives data from other devices or transmitsdata generated by the CPU 1100 to other devices via the communicationinterface 1500.

The input/output interface 1600 is an interface for connecting betweenan input/output device 1650 and the computer 1000. For example, the CPU1100 receives data from an input device such as a keyboard or a mousevia the input/output interface 1600. In addition, the CPU 1100 transmitsdata to an output device such as a display, a speaker, or a printer viathe input/output interface 1600. Furthermore, the input/output interface1600 may function as a media interface for reading a program or the likerecorded on predetermined recording medium (or simply medium). Examplesof the media include optical recording media such as a digital versatiledisc (DVD), a magneto-optical recording medium such as a magneto-opticaldisk (MO), a tape medium, a magnetic recording medium, and semiconductormemory.

For example, in a case where the computer 1000 functions as theinformation processing device 30 according to the embodiment, the CPU1100 of the computer 1000 executes the program loaded on the RAM 1200,thereby implementing the functions of the control unit 32, namely,functions of the measurement unit 321, the first recognition unit 322,the second recognition unit 323, the missing defect detection unit 324,the estimation unit 325, the specifying unit 326, the processing unit327, the display control unit 328, and the like. The HDD 1400 stores theprogram and the data in the storage unit 31 according to the presentdisclosure. While the CPU 1100 executes the program data 1450 read fromthe HDD 1400, the CPU 1100 may acquire these programs from anotherdevice via the external network 1550, as another example.

The preferred embodiments of the present disclosure have been describedin detail above with reference to the accompanying drawings. However,the technical scope of the present disclosure is not limited to suchexamples. It will be apparent to those skilled in the art of the presentdisclosure that various modifications and alterations can be conceivedwithin the scope of the technical idea described in the claims andnaturally fall within the technical scope of the present disclosure.

Furthermore, the effects described in the present specification aremerely illustrative or exemplary and are not limited. That is, thetechnique according to the present disclosure can exhibit other effectsthat are apparent to those skilled in the art from the description ofthe present specification in addition to or instead of the aboveeffects.

It is also possible to create a program for the hardware such as a CPU,ROM, and RAM built in a computer to exert the functions equivalent tothe configuration of the information processing device 30, and acomputer-readable recording medium that has recorded the program canalso be provided.

Furthermore, individual steps related to the processing of theinformation processing device 30 in the present specification do notnecessarily have to be processed in chronological order in the orderdescribed in the flowchart. For example, individual steps related to theprocessing of the information processing device 30 may be processed inan order different from the order described in the flowchart, or may beprocessed in parallel.

(Effects)

The information processing device 30 includes: the storage unit 31 thatstores the first information 31A indicating at least one of a structureor a physical property of a first object obtained by capturing a realphysical body into a virtual space; and the specifying unit 326 thatspecifies an arrangement of a second object indicating a virtual objectin the virtual space V so that the arrangement is capable of expressingan interaction with the first object based on the arrangement condition31C of the second object and on the first information 31A.

With this configuration, by arranging the second object at a positionwhere the second object can interact with the first object in thevirtual space V into which the real environment has been captured, theinformation processing device 30 can suppress a gap occurring betweenthe first object and the second object in the virtual space V. As aresult, the information processing device 30 can suppress thestrangeness or incompatibility of the second object displayed in thevirtual space V into which the measured real environment has beencaptured.

The specifying unit 326 of the information processing device 30specifies an arrangement of the second object in the virtual space V sothat the arrangement satisfies physical conditions between a part of thesecond object and the first object corresponding to the arrangementcondition 31C and so that the arrangement is capable of expressing theinteraction with the first object.

With this configuration, the information processing device 30 specifiesthe arrangement of the second object in the virtual space V so as tosatisfy the physical condition between the part of the second object andthe first object, making it possible to arrange the second object at anappropriate position where the interaction can be expressed in thevirtual space V. As a result, the information processing device 30 canachieve natural expression of the interaction between the first objectand the second object in the virtual space V.

The specifying unit 326 of the information processing device 30specifies an arrangement of the second object in the virtual space V sothat the arrangement is capable of expressing the interaction with thefirst object based on the positional relationship of the components ofthe first object in the virtual space V.

With this configuration, the information processing device 30 specifiesthe arrangement of the second object in the virtual space V inconsideration of the positional relationship of the components of thefirst object in the virtual space V, making it possible to arrange thesecond object at an appropriate position in the virtual space V wherethe interaction can be expressed. As a result, the informationprocessing device 30 can arrange the second object in the componentcapable of expressing the interaction by the first object in the virtualspace V, making it possible to achieve more natural expression of theinteraction.

The specifying unit 326 of the information processing device 30specifies an arrangement of the second object in the virtual space V sothat the arrangement is capable of expressing the interaction with thefirst object based on the second information 31B indicating thecharacteristic of the second object, the arrangement condition 31C, andthe first information 31A.

With this configuration, the information processing device 30 specifiesthe arrangement of the object C in the virtual space V in considerationof the characteristic of the second object in the virtual space V,making it possible to arrange the second object at an appropriateposition in the virtual space V suitable for the characteristic of thesecond object and capable of expressing the interaction. As a result,the information processing device 30 can achieve natural expression ofthe interaction between the first object and the second object in thevirtual space V without interfering with the expression of thecharacteristic of the second object in the virtual space V.

In the information processing device 30, in a case where it is notpossible to specify an arrangement of the second object in the virtualspace V so that the arrangement is capable of expressing the interactionwith the first object, the specifying unit 326 specifies the arrangementof the second object based on another arrangement condition 31Cdifferent from the arrangement condition 31C.

With this operation, in a case where the second object cannot bearranged in the virtual space V so as to satisfy the arrangementcondition 31C, the information processing device 30 can specify thearrangement of the second object so as to satisfy another arrangementcondition 31C in the virtual space V. As a result, the informationprocessing device 30 can prevent an occurrence of unnatural expressionof the interaction between the first object and the second object in thevirtual space V.

In the information processing device 30, in a case where the specifyingunit 326 cannot specify an arrangement of the second object in thevirtual space V so that the arrangement is capable of expressing theinteraction with the first object, the second object is not to bearranged in the virtual space V.

With this operation, in a case where the second object cannot bearranged in the virtual space V so as to satisfy the arrangementcondition 31C, the information processing device 30 can stop thearrangement of the second object in the virtual space V. As a result,the information processing device 30 can prevent an occurrence ofunnatural expression of the interaction between the first object and thesecond object in the virtual space V.

The information processing device 30 further includes the processingunit 327 that executes the process of expressing an interaction betweenthe first object and the second object based on the arrangement of thesecond object in the virtual space V specified by the specifying unit326 and on the first information 31A.

With this configuration, the information processing device 30 canexpress an interaction between the first object and the second objectbased on the arrangement of the second object in the virtual space V andthe structure and physical properties of the first object. As a result,the information processing device 30 can express the interaction withthe second object in consideration of the structure and physicalproperties of the first object, making it possible to further suppressthe strangeness or incompatibility of the second object displayed in thevirtual space V.

The information processing device 30 further includes the secondrecognition unit 323 that generates the first information 31A based onthe recognition result of recognizing the real physical body and on theinformation indicating the structure and physical properties, and thestorage unit 31 stores the first information 31A generated by the secondrecognition unit 323.

With this configuration, the information processing device 30 cangenerate the first information 31A indicating the structure and physicalproperties of the first object obtained by capturing the real physicalbody into the virtual space V, and can store the first information 31Ain the storage unit 31. As a result, by generating the first information31A of the physical body captured in the virtual space V, theinformation processing device 30 can achieve natural expression of theinteraction between the first object and the second object.

The information processing device 30 further includes: the missingdefect detection unit 324 that detects a missing defect of a realphysical body based on the first information 31A generated by the secondrecognition unit 323; and the missing defect complementing unit 324Athat changes the first information 31A to complement a missing defect ina real physical body when the missing defect has been detected, and thestorage unit 31 stores the first information 31A changed by the missingdefect complementing unit 324A.

With this configuration, when having detected a missing defect in a realphysical body based on the first information 31A, the informationprocessing device 30 can change the first information 31A to complementthe missing defect. As a result, by complementing the missing defect ofthe physical body captured into the virtual space V, the informationprocessing device 30 can achieve more natural expression of theinteraction between the first object and the second object.

The information processing device 30 further includes: the estimationunit 325 that estimates an interaction between the plurality of firstobjects based on the first information 31A generated by the secondrecognition unit 323; and the correction unit 325A that corrects thefirst information 31A based on the estimation result of the estimationunit 325, and the storage unit 31 stores the first information 31Acorrected by the correction unit 325A.

With this configuration, the information processing device 30 canestimate the interaction between the plurality of first objects based onthe first information 31A, and can correct the first information 31Abased on the estimation result. As a result, by correcting the firstinformation 31A based on the interaction between the physical bodiescaptured in the virtual space V, the information processing device 30can achieve more natural expression of the interaction between the firstobject and the second object.

The processing unit 327 of the information processing device 30 executesa process of expressing an interaction occurring between the pluralityof first objects in the virtual space V based on the plurality of piecesof first information 31A and the motion of the second object.

With this configuration, the information processing device 30 canexpress an interaction between the plurality of first objects based onthe motion of the second object in the virtual space V and the structureand physical properties of the first object. As a result, theinformation processing device 30 can express the interaction between theplurality of first objects obtained by capturing the first objects intothe virtual space V, making it possible to achieve natural expression ofthe first objects displayed in the virtual space V.

The specifying unit 326 of the information processing device 30specifies the arrangement of the third object operable by the secondobject in the virtual space V based on the arrangement condition of thethird object and on the first information 31A, and the processing unit327 executes a process of expressing the interaction occurring betweenthe third object and the first object based on a characteristic of thethird object and on the first information 31A.

With this configuration, the information processing device 30 canexpress an interaction between the third object and the first objectbased on the third object in the virtual space V and the structure andphysical properties of the first object. As a result, the informationprocessing device 30 can express the interaction between the thirdobject operable by the second object, and the first object, making itpossible to achieve natural expression of the third object displayed inthe virtual space V.

An information processing method is a method to be executed by acomputer, and the method includes: storing, in the storage unit 31, thefirst information 31A indicating at least one of a structure or aphysical property of a first object obtained by capturing a realphysical body into a virtual space; and specifying an arrangement of asecond object indicating a virtual object in the virtual space V so thatthe arrangement is capable of expressing an interaction with the firstobject, based on the arrangement condition 31C of the second object andon the first information 31A.

With this configuration, by arranging, by the computer, the secondobject at a position where the second object can interact with the firstobject in the virtual space V into which the real environment has beencaptured, the information processing method is capable of suppressing agap occurring between the first object and the second object in thevirtual space V. As a result, the information processing method iscapable of suppressing the strangeness or incompatibility of the secondobject displayed in the virtual space V into which the measured realenvironment has been captured.

A program causes a computer to execute processes including: storing, inthe storage unit 31, the first information 31A indicating at least oneof a structure or a physical property of a first object obtained bycapturing a real physical body into a virtual space; and specifying anarrangement of a second object indicating a virtual object in thevirtual space V so that the arrangement is capable of expressing aninteraction with the first object, based on the arrangement condition31C of the second object and on the first information 31A.

With this processes, the program can cause the computer to arrange thesecond object at a position where the second object can interact withthe first object in the virtual space V into which the real environmenthas been captured, making it possible to suppress a gap generatedbetween the first object and the second object in the virtual space V.As a result, the program is capable of suppressing the strangeness orincompatibility of the second object displayed in the virtual space Vinto which the measured real environment has been captured.

Note that the following configurations also belong to the technicalscope of the present disclosure.

(1)

An information processing device comprising:

a storage unit that stores first information indicating at least one ofa structure or a physical property of a first object obtained bycapturing a real physical body into a virtual space; and

a specifying unit that specifies an arrangement of a second objectindicating a virtual object in the virtual space so that the arrangementis capable of expressing an interaction with the first object, based onan arrangement condition of the second object and on the firstinformation.

(2)

The information processing device according to (1),

wherein the specifying unit specifies the arrangement of the secondobject in the virtual space so that the arrangement satisfies physicalconditions between a part of the second object and the first objectcorresponding to the arrangement condition and so that the arrangementis capable of expressing an interaction with the first object.

(3)

The information processing device according to (1) or (2),

wherein the specifying unit specifies the arrangement of the secondobject in the virtual space so that the arrangement is capable ofexpressing an interaction with the first object based on a positionalrelationship of components of the first object in the virtual space.

(4)

The information processing device according to any one of (1) to (3),

wherein the specifying unit specifies the arrangement of the secondobject in the virtual space so that the arrangement is capable ofexpressing an interaction with the first object based on secondinformation indicating a characteristic of the second object, thearrangement condition, and the first information.

(5)

The information processing device according to any one of (1) to (4),

wherein, when it is not possible to specify the arrangement of thesecond object in the virtual space so that the arrangement is capable ofexpressing an interaction with the first object, the specifying unitspecifies the arrangement of the second object based on anotherarrangement condition different from the arrangement condition.

(6)

The information processing device according to any one of (1) to (4),

wherein, when it is not possible to specify the arrangement of thesecond object in the virtual space so that the arrangement is capable ofexpressing an interaction with the first object, the specifying unitdoes not arrange the second object in the virtual space.

(7)

The information processing device according to any one of (1) to (6),further comprising

a processing unit that executes a process of expressing an interactionbetween the first object and the second object based on the arrangementof the second object in the virtual space having been specified by thespecifying unit and on the first information.

(8)

The information processing device according to any one of (1) to (7),further comprising

a recognition unit that generates the first information based on arecognition result obtained by recognizing a real physical body and oninformation indicating a structure and physical properties,

wherein the storage unit stores the first information generated by therecognition unit.

(9)

The information processing device according to (8), further comprising:

a detection unit that detects a missing defect of the real physical bodybased on the first information generated by the recognition unit; and

a complementing unit that changes the first information so as tocomplement a missing defect of the real physical body when the missingdefect of the real physical body has been detected,

wherein the storage unit stores the first information changed by thecomplementing unit.

(10)

The information processing device according to (8) or (9), furthercomprising:

an estimation unit that estimates an interaction between a plurality ofthe first objects based on the first information generated by therecognition unit; and

a correction unit that corrects the first information based on anestimation result of the estimation unit,

wherein the storage unit stores the first information corrected by thecorrection unit.

(11)

The information processing device according to (7),

wherein the processing unit executes a process of expressing aninteraction occurring between a plurality of the first objects in thevirtual space based on a plurality of pieces of the first informationand a motion of the second object.(12)

The information processing device according to (7),

wherein the specifying unit specifies an arrangement of a third objectoperable by the second object in the virtual space based on anarrangement condition of the third object and on the first information,and

the processing unit executes a process of expressing an interactionoccurring between the third object and the first object based on acharacteristic of the third object and on the first information.

(13)

An information processing method executed by a computer, the methodcomprising:

storing, in a storage unit that stores first information indicating atleast one of a structure or a physical property of a first objectobtained by capturing a real physical body into a virtual space; and

specifying an arrangement of a second object indicating a virtual objectin the virtual space so that the arrangement is capable of expressing aninteraction with the first object, based on an arrangement condition ofthe second object and on the first information.

(14)

A program for causing a computer to execute:

storing, in a storage unit, first information indicating at least one ofa structure or a physical property of a first object obtained bycapturing a real physical body into a virtual space; and

specifying an arrangement of a second object indicating a virtual objectin the virtual space so that the arrangement is capable of expressing aninteraction with the first object, based on an arrangement condition ofthe second object and on the first information.

REFERENCE SIGNS LIST

-   -   10 SENSOR UNIT    -   20 DISPLAY DEVICE    -   30 INFORMATION PROCESSING DEVICE    -   31A FIRST INFORMATION    -   31B SECOND INFORMATION    -   31C ARRANGEMENT CONDITION    -   31M MAP INFORMATION    -   32 CONTROL UNIT    -   321 MEASUREMENT UNIT    -   322 FIRST RECOGNITION UNIT    -   323 SECOND RECOGNITION UNIT    -   324 MISSING DEFECT DETECTION UNIT    -   324A MISSING DEFECT COMPLEMENTING UNIT    -   325 ESTIMATION UNIT    -   325A CORRECTION UNIT    -   326 SPECIFYING UNIT    -   327 PROCESSING UNIT    -   328 DISPLAY CONTROL UNIT    -   C OBJECT    -   P REAL ENVIRONMENT    -   R PHYSICAL BODY OBJECT    -   V VIRTUAL SPACE

1. An information processing device comprising: a storage unit thatstores first information indicating at least one of a structure or aphysical property of a first object obtained by capturing a realphysical body into a virtual space; and a specifying unit that specifiesan arrangement of a second object indicating a virtual object in thevirtual space so that the arrangement is capable of expressing aninteraction with the first object, based on an arrangement condition ofthe second object and on the first information.
 2. The informationprocessing device according to claim 1, wherein the specifying unitspecifies the arrangement of the second object in the virtual space sothat the arrangement satisfies physical conditions between a part of thesecond object and the first object corresponding to the arrangementcondition and so that the arrangement is capable of expressing aninteraction with the first object.
 3. The information processing deviceaccording to claim 2, wherein the specifying unit specifies thearrangement of the second object in the virtual space so that thearrangement is capable of expressing an interaction with the firstobject based on a positional relationship of components of the firstobject in the virtual space.
 4. The information processing deviceaccording to claim 3, wherein the specifying unit specifies thearrangement of the second object in the virtual space so that thearrangement is capable of expressing an interaction with the firstobject based on second information indicating a characteristic of thesecond object, the arrangement condition, and the first information. 5.The information processing device according to claim 4, wherein, when itis not possible to specify the arrangement of the second object in thevirtual space so that the arrangement is capable of expressing aninteraction with the first object, the specifying unit specifies thearrangement of the second object based on another arrangement conditiondifferent from the arrangement condition.
 6. The information processingdevice according to claim 4, wherein, when it is not possible to specifythe arrangement of the second object in the virtual space so that thearrangement is capable of expressing an interaction with the firstobject, the specifying unit does not arrange the second object in thevirtual space.
 7. The information processing device according to claim1, further comprising a processing unit that executes a process ofexpressing an interaction between the first object and the second objectbased on the arrangement of the second object in the virtual spacehaving been specified by the specifying unit and on the firstinformation.
 8. The information processing device according to claim 1,further comprising a recognition unit that generates the firstinformation based on a recognition result obtained by recognizing a realphysical body and on information indicating a structure and physicalproperties, wherein the storage unit stores the first informationgenerated by the recognition unit.
 9. The information processing deviceaccording to claim 8, further comprising: a detection unit that detectsa missing defect of the real physical body based on the firstinformation generated by the recognition unit; and a complementing unitthat changes the first information so as to complement a missing defectof the real physical body when the missing defect of the real physicalbody has been detected, wherein the storage unit stores the firstinformation changed by the complementing unit.
 10. The informationprocessing device according to claim 8, further comprising: anestimation unit that estimates an interaction between a plurality of thefirst objects based on the first information generated by therecognition unit; and a correction unit that corrects the firstinformation based on an estimation result of the estimation unit,wherein the storage unit stores the first information corrected by thecorrection unit.
 11. The information processing device according toclaim 7, wherein the processing unit executes a process of expressing aninteraction occurring between a plurality of the first objects in thevirtual space based on a plurality of pieces of the first informationand a motion of the second object.
 12. The information processing deviceaccording to claim 7, wherein the specifying unit specifies anarrangement of a third object operable by the second object in thevirtual space based on an arrangement condition of the third object andon the first information, and the processing unit executes a process ofexpressing an interaction occurring between the third object and thefirst object based on a characteristic of the third object and on thefirst information.
 13. An information processing method executed by acomputer, the method comprising: storing, in a storage unit that storesfirst information indicating at least one of a structure or a physicalproperty of a first object obtained by capturing a real physical bodyinto a virtual space; and specifying an arrangement of a second objectindicating a virtual object in the virtual space so that the arrangementis capable of expressing an interaction with the first object, based onan arrangement condition of the second object and on the firstinformation.
 14. A program for causing a computer to execute: storing,in a storage unit, first information indicating at least one of astructure or a physical property of a first object obtained by capturinga real physical body into a virtual space; and specifying an arrangementof a second object indicating a virtual object in the virtual space sothat the arrangement is capable of expressing an interaction with thefirst object, based on an arrangement condition of the second object andon the first information.